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(3) Detecting scene cuts in video processing. 

(57) A method for detecting scene cuts in a motion picture, which comprises a plurality of frames in a 
sequential order. The preferred embodiment of the present invention utilizes the correlation between 
successive frames to detect abrupt cuts. Gradual cuts are preferably detected by examining a function 
consisting of the standard deviation of the pixels of each frame as a function of the frame number. 



LU 



Jouve, 18, rue Saint-Denis, 75001 PARIS 



♦ 




EP 0 637 027 A2 



The present invention relates to video processing and more particularly, to an improved method for auto- 
matically detecting the start of a new scene in a video sequence. 

Atypical motion picture consists of a sequence of scenes. Within a scene, each frame differs only slightly 
from the preceding frame. The change from one scene to the next will be referred to as "cut" in the following 
5 discussion. Scene cuts may be abrupt or gradual. Gradual scene cuts are associated with special effects such 
as camera pans, dissolves, fades to black or fade from black. 

One method for summarizing a motion picture is to provide a log of the scenes. A scene log is a list of the 
scenes and the location of the start of the scene from some reference point such as the start of the motion 
picture. Atypical frame from the scene may be used to summarize the scene. 
w Among other things, such logs are useful in video editing. The log provides a means for an editor to quickly 
find a particular scene. 

Prior art schemes for automating the logging of motion pictures rely on the difference between sequential 
frames to determine where a cut has occurred. These methods compute the average value of some parameter 
such as the absolute value of the difference of corresponding pixels in two sequential frames. Such methods 
15 do not detect gradual cuts with sufficient reliability to be useful in automating the cut logging process. In ad- 
dition, there are some situations in which even abrupt cuts are not detected by these methods. 

Broadly, it is the object of the present invention to provide an improved method for detecting scene cuts. 

It is a further object of the present invention to provide an automated method for detecting both gradual 
scene and abrupt cuts. 

20 Accordingly, the invention proposes a method for detecting scene cuts in a motion picture, said motion 
picture comprising an ordered sequence of frames, each said frame comprising a plurality of intensity values, 
said method comprising the steps of determining a correlation value of each frame with a corresponding pre- 
vious frame, said previous frame being separated in said sequence by a predetermined number of frames; 
comparing said correlation value with a threshold value; and reporting an abrupt scene change if said corre- 

25 lation value is less than said threshold value. 

The preferred embodiment of the present invention utilizes the correlation between successive frames to 
detect abrupt cuts. Gradual cuts are preferably detected by examining a function consisting of the standard 
deviation of the pixels of each frame as a function of the frame number. Gradual cuts are characterized by 
excursions in this function. A gradual scene cut may either manifest itself by a peak in this function or by a 

30 ramp connecting two regions of relatively constant values. 

Figure 1 is a flow chart of an exemplary method to the present invention for finding the cuts in a motion 
picture. 

The present embodiment utilizes the correlation of the pixels in successive frames to detect abrupt scene 
cuts. The method by which more gradual scene cuts are detected will be explained in more detail below. Denote 
35 the pixels of the nth frame of a motion picture by n l(x,y). That is n l(x,y) is the intensity of the pixel at some point 
(x,y) in the frame. For the purposes of this discussion, it will be assumed that n l(x,y) is defined on a regular 
grid having a total of N grid points. For two frames, (n-k) and n, the method of the present invention computes 
a correlation factor, p n , k , where 

Pnk = E[< n - k >l(x,y)'"l(x,y)]- E[("-*)l(x,y)1 * EPI(x,y)1 

<*n * <*(n-1) 

where 
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E[0(x,y)] = -ill"l(x,y) (2) 



and 



a n = VE["|2(x,y)]-E2[n|(x,y)] (3) 
In the preferred embodiment of the present invention, k=1 . Within any given scene, p„ t1 is expected to be near 
one. If an abrupt scene cut occurs at frame n, then p ni1 decrease to near zero. The present invention tracks 
the value of p Ptl from frame to frame and defines an abrupt scene cut as having occurred at frame n if p n ,i is 
below some predetermined value. 

In practice, the threshold value may be found by examining a few scene changes. However, it will be ap- 
parent to those skilled in the art that fixed thresholds may be used. The cut-off can also be defined in terms 
of the previously observed values of p n>1 . In such a system, an abrupt scene cut would be defined to have oc- 
curred at frame n if p n ,i is less than the average value of p n ,i over the previous M frames where M is some 
integer greater than 0. 
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While the above method finds abrupt scene cuts with a high degree of reliability, it is not a reliable method 
for finding gradual scene cuts, since the amount of change between successive frames in a gradual cut is too 
small to substantially change the value of p n>1 . In principle, the method discussed above can be extended to 
find gradual scene cuts by increasing the "distance" between the frames for which P n ,k is computed, i.e., choos- 
5 ing k>1. In such a system, abrupt cuts would be detected using p n>1 , and gradual cuts would be detected using 
p nik with k set to some predetermined value. While such a scheme is capable of detecting more of the gradual 
cuts, it requires some knowledge of the correct value for k. In addition, further analysis is needed to determine 
the precise location of the cut 

In the preferred embodiment of the present invention, an alternative method is used to detect gradual cuts. 
10 As discussed above, the computation of p n>k requires the computation of a n . The preferred embodiment of the 
present invention uses the behavior of a n to detect gradual scene cuts. The value of cr n for any frame measures 
the variation of the pixel values about the mean pixel value in the frame. Within any given scene, the values 
of a n from frame to frame remain roughly constant. When a scene slowly changes, the value <r n changes from 
the value representative of the previous scene to the value representative of the new scene. This change takes 
15 place over a number of frames. 

The specific manner in which a n changes will depend on the starting and ending scene, and the nature 
of the special effect. For example, in the case of a fade to black followed by a fade from black to the new scene, 
<r n would be expected to decrease during the fade to black and then increase to the value representative of 
the new frame. In practice, the <y n curve has considerably more noise than the p n , k curve; hence, the curve is 
20 smoothed in the preferred embodiment of the present invention to allow automated determination of the gradual 
cuts. In the preferred embodiment of the present invention, the previous 4 points are averaged to generate 
the smoothed version of <r n . 

The excursions of the a n curve during a gradual scene cut are of two possible types. In the first type, the 
CT n curve goes through a peak. This is similar to the fade to black effect discussed above. In the second type, 
25 the c n curve exhibits a "ramp" connecting the two regions of fairly constant o n values. 

A gradual cut is defined as having begun when the absolute value of the slope of the smoothed a n curve 
increases beyond a predetermined threshold value. The increased slope value must be present for a prede- 
termined first time interval, i.e., number of frames, before the gradual cut is regarded as valid. When the ab- 
solute value of the slope has remained below a predetermined threshold for a second time interval, the gradual 
30 cut is deemed to have been completed. Between the beginning and end of the gradual cut, the slope of the a n 
curve will, in general, go to zero briefly as the a n curve passes through an inflection point if a peak is present; 
hence, the gradual cut is not defined to have been completed until the absolute value of the slope of the o n 
curve remains below the threshold for some predetermined time interval. 

Refer now to Figure 1 which is a flow chart of a scene eut detecting program according to the present in- 
35 vention. In the preferred embodiment of the present invention, the value of k in Eq. (1) is set to one. To simplify 
the notation, p n>1 has been replaced by p in the following discussion. After initializing various variables as shown 
at 10, the program acquires the next frame of the motion picture and computes the values for p and the average 
value of a as shown at 12. The program then tests the value of p to determine if an abrupt cut has occurred 
as shown at 14. An abrupt cut is deemed to have occurred if the value of p is less than a predetermined thresh- 
40 old. It is possible for an abrupt cut to take more than one frame. Hence, a flag is set as shown at 16 to indicate 
that p has crossed the threshold. The flag will be reset when p once again returns to a value above the thresh- 
old. This flag is used by blocks 18 and 20 to determine if the current value of p being less than the threshold 
is a new cut or just a continuation of the previously detected abrupt cut. If the cut is completed, the flag is reset 
as shown at 22. 

45 If no abrupt cut is in progress, the program goes on to examine the data to determine if a gradual cut is in 

progress. As noted above, the program uses the average value of a to detect gradual cuts. The curve consisting 
of the average value of a as a function of the frame number will be referred to as the a curve. The program 
also keeps track of the slope of the a curve. Changes in the slope of the a curve are used to detect possible 
events in traversing a gradual cut. As noted above, a gradual cut may be evidenced either by a peak in the a 

50 curve or by a ramp connecting two a values representative of a in the frames on each side of the gradual cut. 
In the later case, there will be no change in the sign of the slope of the a curve during the gradual cut 

Referring again to Figure 1, if no abrupt cut is in progress, the program tests for a gradual cut being in 
progress as shown at 24. If a gradual cut is not in progress, the program checks for the beginning of a gradual 
cut as shown at 26. A gradual cut is deemed to have begun if the slope of the a curve has changed and the 

55 change in a is greater than a predetermined threshold value. If these conditions are met, a set of flags is ini- 
tialized as shown at 28. The flag values indicate that a gradual cut has begun and no peak has yet been detected 
in the a curve. 

If a gradual cut was already in progress as indicated by the flags set at 28, the program determines if the 
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current frame is the end of the gradual cut as shown at 30. If the sign of the slope of the a curve has changed, 
the program checks to see if the program has already passed through an inflection point as indicated by one 
of the above-described flags having a value different from its initialization value. If the initial value is still pres- 
ent, the program assumes that the change in sign is due to the presence of an inflection point in the a curve. 

5 The flag is set accordingly as shown at 32. If the a curve has already passed through an inflection point, the 
program assumes that the gradual cut has been completed. 

If the sign of the slope of the a curve has not changed but the change in the a curve from the last frame 
is less than some predetermined threshold, the program assumes that the gradual cut has been completed, 
provided the excursions in the a curve from one scene to the next were sufficiently large. This case is en- 

10 countered when there is no peak in the a curve between the scenes. 

The a curve, even with smoothing, has a significant amount of noise. To distinguish a "ramp" between 
two values in the a curve from noise, the program keeps track of certain reference values on the a curve. The 
value of the a curve at the potential beginning of a ramp is stored. At the end of the ramp, the difference be- 
tween the ending value and the reference values is used to eliminate "noise events". 

15 While the preferred embodiment of the present invention uses the standard deviation of the pixel values 
in each frame for detecting gradual cuts, it will be apparent to those skilled in the art that other statistical meas- 
ures or functions thereof may be used. In this regard, it should be noted that a is a function of the variance of 
the pixel values which in turn is the average value of the second moment of the pixel values. Hence, any func- 
tion based on one of the moments of the pixel distribution may also be utilized. 

20 While the above-described embodiments of the present invention compute the correlation and standard 
deviation using all of the data in each frame, it will be apparent to those skilled in the art that a subset of the 
pixel values may be used to determine these parameters. Any representative subset may be used. Embodi- 
ments based on such subsets generate a lower computational load; however, such embodiments are expected 
to yield a values having more "noise". 

25 The present invention is preferably practiced on a general purpose computer. However, it will be apparent 
to those skilled in the art that special purpose hardware may also be utilized to improve the speed with which 
the method operates. For example, hardware that computes the correlation of two frames and the variance of 
the pixels in a given frame can substantially improve the speed with which the method of the present invention 
operates. 

30 

Claims 

1. A method for detecting scene cuts in a motion picture, said motion picture comprising an ordered se- 
35 quence of frames, each said frame comprising a plurality of intensity values, said method comprising the 

steps of determining a correlation value of each frame with a corresponding previous frame, said previous 
frame being separated in said sequence by a predetermined numberof frames; comparing said correlation 
value with a threshold value; and reporting an abrupt scene change if said correlation value is less than 
said threshold value. 

40 

2. The method of Claim 1 wherein said predetermined number of frames is equal to one. 

3. The method of Claim 1 further comprising the steps of 

determining a statistical measure for each said frame; and 
45 detecting changes in the value of said statistical measure as a function of the frame number and; 

reporting a gradual scene change if said detected changes match a predetermined pattern. 

4. The method of Claim 3 wherein said statistical measure is a function of the variance of said intensity val- 
ues. 

50 5. The method of Claim 3 wherein said statistical measure is a moment of said intensity values. 

6. The method of Claim 3 wherein said predetermined pattern is a peak in said statistical measure as a func- 
tion of the frame number. 

55 7. The method of Claim 3 wherein said predetermined pattern is a ramp connecting two regions in said func- 
tion, said regions of a variation of said statistical measure less than the difference in said statistical meas- 
ure between the beginning and end of said ramp. 
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